Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation

摘要

Convolutional neural networks have enabled accurate image super-resolution inreal-time. However, recent attempts to benefit from temporal correlations invideo super-resolution have been limited to naive or inefficient architectures.In this paper, we introduce spatio-temporal sub-pixel convolution networks thateffectively exploit temporal redundancies and improve reconstruction accuracywhile maintaining real-time speed. Specifically, we discuss the use of earlyfusion, slow fusion and 3D convolutions for the joint processing of multipleconsecutive video frames. We also propose a novel joint motion compensation andvideo super-resolution algorithm that is orders of magnitude more efficientthan competing methods, relying on a fast multi-resolution spatial transformermodule that is end-to-end trainable. These contributions provide both higheraccuracy and temporally more consistent videos, which we confirm qualitativelyand quantitatively. Relative to single-frame models, spatio-temporal networkscan either reduce the computational cost by 30% whilst maintaining the samequality or provide a 0.2dB gain for a similar computational cost. Results onpublicly available datasets demonstrate that the proposed algorithms surpasscurrent state-of-the-art performance in both accuracy and efficiency.